Asymptotic minimaxity of False Discovery Rate thresholding for sparse exponential data

We apply FDR thresholding to a non-Gaussian vector whose coordinates X_i, i=1,..., n, are independent exponential with individual means $\mu_i$. The vector $\mu =(\mu_i)$ is thought to be sparse, with most coordinates 1 but a small fraction significantly larger than 1; roughly, most coordinates are simply `noise,' but a small fraction contain `signal.' We measure risk by per-coordinate mean-squared error in recovering $\log(\mu_i)$, and study minimax estimation over parameter spaces defined by constraints on the per-coordinate p-norm of $\log(\mu_i)$: $\frac{1}{n}\sum_{i=1}^n\log^p(\mu_i)\leq \eta^p$. We show for large n and small $\eta$ that FDR thresholding can be nearly Minimax. The FDR control parameter 0<q<1 plays an important role: when $q\leq 1/2$, the FDR estimator is nearly minimax, while choosing a fixed q>1/2 prevents near minimaxity. These conclusions mirror those found in the Gaussian case in Abramovich et al. [Ann. Statist. 34 (2006) 584--653]. The techniques developed here seem applicable to a wide range of other distributional assumptions, other loss measures and non-i.i.d. dependency structures.Comment: Published at http://dx.doi.org/10.1214/009053606000000920 in the Annals of Statistics (http://www.imstat.org/aos/) by the Institute of Mathematical Statistics (http://www.imstat.org

Higher criticism for detecting sparse heterogeneous mixtures

Higher criticism, or second-level significance testing, is a multiple-comparisons concept mentioned in passing by Tukey. It concerns a situation where there are many independent tests of significance and one is interested in rejecting the joint null hypothesis. Tukey suggested comparing the fraction of observed significances at a given \alpha-level to the expected fraction under the joint null. In fact, he suggested standardizing the difference of the two quantities and forming a z-score; the resulting z-score tests the significance of the body of significance tests. We consider a generalization, where we maximize this z-score over a range of significance levels 0<\alpha\leq\alpha_0. We are able to show that the resulting higher criticism statistic is effective at resolving a very subtle testing problem: testing whether n normal means are all zero versus the alternative that a small fraction is nonzero. The subtlety of this ``sparse normal means'' testing problem can be seen from work of Ingster and Jin, who studied such problems in great detail. In their studies, they identified an interesting range of cases where the small fraction of nonzero means is so small that the alternative hypothesis exhibits little noticeable effect on the distribution of the p-values either for the bulk of the tests or for the few most highly significant tests. In this range, when the amplitude of nonzero means is calibrated with the fraction of nonzero means, the likelihood ratio test for a precisely specified alternative would still succeed in separating the two hypotheses.Comment: Published by the Institute of Mathematical Statistics (http://www.imstat.org) in the Annals of Statistics (http://www.imstat.org/aos/) at http://dx.doi.org/10.1214/00905360400000026

Counting faces of randomly-projected polytopes when the projection radically lowers dimension

This paper develops asymptotic methods to count faces of random high-dimensional polytopes. Beyond its intrinsic interest, our conclusions have surprising implications - in statistics, probability, information theory, and signal processing - with potential impacts in practical subjects like medical imaging and digital communications. Three such implications concern: convex hulls of Gaussian point clouds, signal recovery from random projections, and how many gross errors can be efficiently corrected from Gaussian error correcting codes.Comment: 56 page

Continuous Curvelet Transform: I. Resolution of the Wavefront Set

We discuss a Continuous Curvelet Transform (CCT), a transform f → Γf (a, b, θ) of functions f(x1, x2) on R^2, into a transform domain with continuous scale a > 0, location b ∈ R^2, and orientation θ ∈ [0, 2π). The transform is defined by Γf (a, b, θ) = {f, γabθ} where the inner products project f onto analyzing elements called curvelets γ_(abθ) which are smooth and of rapid decay away from an a by √a rectangle with minor axis pointing in direction θ. We call them curvelets because this anisotropic behavior allows them to ‘track’ the behavior of singularities along curves. They are continuum scale/space/orientation analogs of the discrete frame of curvelets discussed in Candès and Donoho (2002). We use the CCT to analyze several objects having singularities at points, along lines, and along smooth curves. These examples show that for fixed (x0, θ0), Γf (a, x0, θ0) decays rapidly as a → 0 if f is smooth near x0, or if the singularity of f at x0 is oriented in a different direction than θ_0. Generalizing these examples, we state general theorems showing that decay properties of Γf (a, x0, θ0) for fixed (x0, θ0), as a → 0 can precisely identify the wavefront set and the H^m- wavefront set of a distribution. In effect, the wavefront set of a distribution is the closure of the set of (x0, θ0) near which Γf (a, x, θ) is not of rapid decay as a → 0; the H^m-wavefront set is the closure of those points (x0, θ0) where the ‘directional parabolic square function’ S^m(x, θ) = ( ʃ|Γf (a, x, θ)|^2 ^(da) _a^3+^(2m))^(1/2) is not locally integrable. The CCT is closely related to a continuous transform used by Hart Smith in his study of Fourier Integral Operators. Smith’s transform is based on strict affine parabolic scaling of a single mother wavelet, while for the transform we discuss, the generating wavelet changes (slightly) scale by scale. The CCT can also be compared to the FBI (Fourier-Bros-Iagolnitzer) and Wave Packets (Cordoba-Fefferman) transforms. We describe their similarities and differences in resolving the wavefront set

Continuous Curvelet Transform: II. Discretization and Frames

We develop a unifying perspective on several decompositions exhibiting directional parabolic scaling. In each decomposition, the individual atoms are highly anisotropic at fine scales, with effective support obeying the parabolic scaling principle length ≈ width^2. Our comparisons allow to extend Theorems known for one decomposition to others. We start from a Continuous Curvelet Transform f → Γ_f (a, b, θ) of functions f(x_1, x_2) on R^2, with parameter space indexed by scale a > 0, location b ∈ R^2, and orientation θ. The transform projects f onto a curvelet γ_(abθ), yielding coefficient Γ_f (a, b, θ) = f, _(γabθ); the corresponding curvelet γ_(abθ) is defined by parabolic dilation in polar frequency domain coordinates. We establish a reproducing formula and Parseval relation for the transform, showing that these curvelets provide a continuous tight frame. The CCT is closely related to a continuous transform introduced by Hart Smith in his study of Fourier Integral Operators. Smith’s transform is based on true affine parabolic scaling of a single mother wavelet, while the CCT can only be viewed as true affine parabolic scaling in euclidean coordinates by taking a slightly different mother wavelet at each scale. Smith’s transform, unlike the CCT, does not provide a continuous tight frame. We show that, with the right underlying wavelet in Smith’s transform, the analyzing elements of the two transforms become increasingly similar at increasingly fine scales. We derive a discrete tight frame essentially by sampling the CCT at dyadic intervals in scale a_j = 2^−j, at equispaced intervals in direction, θ_(jℓ), = 2π2^(−j/2)ℓ, and equispaced sampling on a rotated anisotropic grid in space. This frame is a complexification of the ‘Curvelets 2002’ frame constructed by Emmanuel Candès et al. [1, 2, 3]. We compare this discrete frame with a composite system which at coarse scales is the same as this frame but at fine scales is based on sampling Smith’s transform rather than the CCT. We are able to show a very close approximation of the two systems at fine scales, in a strong operator norm sense. Smith’s continuous transform was intended for use in forming molecular decompositions of Fourier Integral Operators (FIO’s). Our results showing close approximation of the curvelet frame by a composite frame using true affine paraboblic scaling at fine scales allow us to cross-apply Smith’s results, proving that the discrete curvelet transform gives sparse representations of FIO’s of order zero. This yields an alternate proof of a recent result of Candès and Demanet about the sparsity of FIO representations in discrete curvelet frames